Our work focuses on providing a mechanism capable of measuring the performance of a MCDM on a given set of alternatives, with the collateral goal of building a global ranking of the e ffectiveness of di fferent MCDMs. We have implemented these tests within the open-source Scikit-Criteria library, leveraging its RankResult and RanksComparator data structures as fundamental building blocks for comparative ranking analysis. RRT1 systematically evaluates the stability of the optimal alternative when suboptimal alternatives are degraded, employing a controlled mutation strategy and providing comprehensive documentation of the experimental context. This approach provides decision analysts with the following: 1. Quantitative stability assessment: Precise measures of how often methods exhibit rank reversal 2. Sensitivity mapping: Identification of which alternatives and criteria are most prone to instability 3. Method comparison: Objective basis for comparing the robustness of di fferent MCDA approaches 4. Confidence intervals: Statistical bounds on decision reliability through repeated experimentation The algorithm addresses the complications that arise from preprocessing pipelines that can eliminate alternatives, ensuring "graceful degradation" by assigning appropriate worst ranks to maintain completeness.

Commonly, AI or machine learning (ML) models are evaluated on benchmark datasets. This practice supports innovative methodological research, but benchmark performance can be poorly correlated with performance in real-world applications -- a construct validity issue. To improve the validity and practical usefulness of evaluations, we propose using an estimands framework adapted from international clinical trials guidelines. This framework provides a systematic structure for inference and reporting in evaluations, emphasizing the importance of a well-defined estimation target. We illustrate our proposal on examples of commonly used evaluation methodologies - involving cross-validation, clustering evaluation, and LLM benchmarking - that can lead to incorrect rankings of competing models (rank reversals) with high probability, even when performance differences are large. We demonstrate how the estimands framework can help uncover underlying issues, their causes, and potential solutions. Ultimately, we believe this framework can improve the validity of evaluations through better-aligned inference, and help decision-makers and model users interpret reported results more effectively.

Cone distribution functions from statistics are turned into Multi-Criteria Decision Making tools. It is demonstrated that this procedure can be considered as an upgrade of the weighted sum scalarization insofar as it absorbs a whole collection of weighted sum scalarizations at once instead of fixing a particular one in advance. As examples show, this type of scalarization--in contrast to a pure weighted sum scalarization-is also able to detect ``non-convex" parts of the Pareto frontier. Situations are characterized in which different types of rank reversal occur, and it is explained why this might even be useful for analyzing the ranking procedure. The ranking functions are then extended to sets providing unary indicators for set preferences which establishes, for the first time, the link between set optimization methods and set-based multi-objective optimization. A potential application in machine learning is outlined.